, I« v «»tore: Richard Meyer et al. l.p'Q Jffa ^ 0 fe . s B;feiqQB 
Title: Failover Processing In A Storage System Jfc» i 

Patent App. No. 10/076,906; Filed: 2/13/02 ^ 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 1 of 37 



Error Reporting 



SNMP 1 1 Alarm~| [Logging I 



I 

^ Report Error 

I 



Error Analysis 



Error Recovery Sequence] 



Error Statistics 



] 



Resource* 



Reliability Rating | 



Error Recovery 
Table 



T 

Request Error ^ 
Analysis ^ 





Take Corrective 
Action 



Report Corrective 
Action Results 





Recovery Management 



Failover 


Failover 
LC 


Failover 
VSX 




Failover 
VSC 


Failover 
SP 



RAID 



Remove RAID 
Member 



Reconstruct 
RAID 
Member 



SP 








Reset 








Dump 








Re-init 














Other ... 

Run 
Diagnostic 



Shutdown 



Reload Code 



Retry | 



Define Error Types and Recovery 
Actions for Resource 

Report Status 



■ Status , 
| Reporting J 




i 



Resource 
State Change 
Commands 



Resource Manager 



Panic 



VSC 



] 



VSX | 



Heartbeat 
Error 







Over- 
Temperature 


The 
Ser 


rmal 
isor 









Processor 
Exception or 
Trap 


L 


c 1 





10 Path Errors"! 



] 



Hardware Error] 



Powerfail | 



Power 
Supply 



Other Errors 



Other 
Resources 



Error 
Collection 




Error Event 



FIG. 1 Error Recovery Architecture 



Inventors: Richard Meyer et akg .Q7 f 'fe O Q . Q fe J. ^ 11 Si 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 2 of 37 



CD 

o> 

CO 
O 
CO 







Adapter 


ll-Ooo 


llQcn 


LL.O-- 


U-Oo 





b r 

-E 

^"3= CO 




CO v- 

c o (3 

-_g -Q -f 

O =1= ^ 



CD 

o 
o 



CO 

o 




CO 
U_ 
i 

o 
CM 

CD 



Inventors: Richard Meyer et alrt QQy & g Q g, , a f J l^g 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 3 of 37 



CD 
CO 
CO 

o 

CO 



1— 












Ada[ 




cd 


















co 

5 




o 





— CO 
^ O CD 



CD 

CD 
CO 







CD 








CO 
< 




CD 









-22 








CO 

3 




CD 









! 55 b 


ies 


Optioi 
nuub 


Switcr 







o 

CO 

=5 



O 
O 



CO 
_CD 
O 



CO 

UL. 

co 



CO 
I 

X 
CO 



cd 

I 

o 

CO 



CO 



CM 



CD 
CO 



O 
CO 



CO 



CXI 



CD 

Dl 
CO 



CD 






CZ 




a 


CO 




CO 













p 


.CD 






% 


_Q 


CO 


Fa 



CD 
CO 
Q- 



CO 

Dl 



o o 

-* — * *EZ 







SP1 


CO 


<sr 




CD 







CD 
i 

o 



CO 



CD 
Q 



CM 



CO ^ 



Q_ 
CO 



CO 



CM 



O 



CD 
Q_ 
CO 



CO 



CM 



^venters: Richard Meyer et al* OfJJjjy }Q 6 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 4 of 37 




Inventors: Richard Meyer et ajMQ Q y w -Q jC, J 

Title: Failover Processing In A Storage System 
, Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 5 of 37 







Failover 










Set (HA) 






Member Composition 


Member 




Member 




Member 


Unit 




Unit 




Unit 



Failover 
Set (FT) 



Composition 



Member 

Y 



vsc 



Member 



vsc 





FIG. 5 Components of a Failover Set 





Unmapped 






Underlying 
Member replaced ^ 


r 






Partially 
Mapped 




Member 
Lost 


Database Replication 
Complete ^ 


f 






Mapped 











FIG. 6 Member P nit State Diagram 



™ Jt mot v Rh * anl I M f? et too? &9Q £» u o s igoe 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 6 of 37 







CD 




> 


a 


O 


.0 


idl 




"c 




CD 


m c 


Q 


"ca 

TD 
CL 


Set 


ZD 







_CD 


o 


"S 


call 


Q_ 

E 


a — 

o 


' Co 



CD 










.52 


"55 




E 


CD 


o 


O 


O 




CD 
CO 

CD 



CO 



CO 
CD 

O 



CD 



t- , ' n , vento p : Richard , M f " et a iq.07 & «3 o & . q 6 3. *s o a 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 7 of 37 




CD 

CD 
CO 

Q 

<D 

-2 

CO 
CD 

E 

CD 



oo 
CD 



Inventors: Richard Meyer e t S * Q Q y ^ g-|] fe „ Qto Q"E 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 8 of 37 



Member-0 


Member-1 


Member-0 Repaired(COLD Boot) 






► 

Member-1 Repaired(COLD Boot) 


-< 

-< 


Member-1 Configured(BACKUP) 


■< 


Member-0 Repaired 


Member-0 Configured(PRIMARY) 




Member-0 COLD Boot and Member-1 
COLD Boot means Assume default 
DB role (e.g., Primary) 


► • 

Member-0 COLD Boot and Member-1 
COLD Boot means Assume default 
DB role (e.g., Backup) 



F I G . 9 Member Arbitration for COLD Boot 



Inventors: Richard Meyer et alrf Q'Q "J , & ^ fj £* 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 9 of 37 



Member-0 


Member-1 


Member-0 Repaired(WARM Boot) 




■< 


► 

Member-1 Repaired(WARM Boot) 


•< 


Member-1 Configured(PRIMARY) 


■< 


Member-0 Repaired 


Member-0 Configured(BACKUP) 


• 


Member-0 WARM Boot and Member-1 
WARM Boot means Assume previous 
DB role (e.g., Backup) 


► 

Member-0 WARM Boot and Member-1 
WARM Boot means Assume previous 
DB role (e.g., Primary) 



FIG. 10 Member Arbitration for WARM Boot 



I^entors: Richard Meyer et als QQ7 £906 ,06I'90E 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 10 of 37 



Member-0 



Member-1 



Member-0 Repaired(COLD Boot) 



Member-1 Repaired(WARM Boot) 



Member-1 Configured(PRIMARY) 



Member-0 Repaired 



Member-0 Configured(BACKUP) 



Member-0 COLD Boot and Member-1 
WARM Boot means Assume remaining 
DB role (e.g., Backup) 



Member-0 COLD Boot and Member-1 
WARM Boot means Assume previous 
DB role (e.g., Primary) 



1 s< , N 

- \ 

' 0 . 



FIG. 11 Member Arbitration for Mixed Boot 



Inventors: Richard Meyer et ata Q & J^ JJ £, ,„ Q £, it «3QHE 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 11 of 37 



CD 



CO 



CO cz 

5 B 

CD O 



2 »~ 

3: -2 

CD O 



CO if 

CD O 
^ < 



— * . . 

CO d 
CD O 



2 s 

CO if 

£ ■£ 



CO c 
CD O 



'co 



7^ co 

CO iz 
CD 13 



CNJ 



CO 



CO if 

5 -2 

CD TS 

2: <c 



oo 

CO fcf 

CD O 
Z < 



00 _ 

co cz* 
CD 13 



CM 



CO 



CD o 



y. en 

CO id 

5 -2 

CD 13 



CZ 

o 
O 



00 Q 

CO cf 

5 -2 

CD 13 



CD -j- 

•I — > . . 

CO t= 

CD O 

"2 <£ 



LO 

ccf 

£ ° 

CO fc£ 

== -g 
CD c5 



CD 
CD 



eo = 

CD 13 



CO c 

5 -2 

CD O 



LO 



CO 



CD <_> 



_CD 

JQ 
CO 
I— 

CD 
CO 

a3 
E 

CD 
CNJ 

CM 

CD 



CD 

*co 

CL 
CD 

en 



CNJ 



DQ 



CO bf 

5 -2 
CD 13 



^ UJ 

CO «= 

5 -2 

CD 13 



co cz 
5 -2 

CD O 



CO b= 
£ -2 

CD O 



CD 
> 
LLI 



CD 

"co 

CD 

a: 



2< 

CO cz 

^ -2 
CD 13 



o 

CO 
QQ 



CO 



co 
E 



CO 



CO 

> 

CO 



iS 
co 

"O 

O 



CD O 



o 

CO 
CQ 



co 
E 

cC 
"co 



CO 

> 

CO 



CNJ 



CO 
CO 



CD O 
~2L <C 



o 

CO 
CQ 



2r 

CO 



Q_ 

CCD 



CO 



CO 

> 

CO 



CO 



o 

CO 
CD 

CO 



CO 



CO 

> 

CO 



CC3 



o 

CO 
CQ 



CO 



CO 

> 



CO 

> 

CO 



CO 
LO 



o 

CO 

CQ 



2r 

CO 

E 



CO 

> 

CO 

> 

CO 



LO 



o 

CO 
CQ 



CO 

> 



CO 

> 

CO 

cz 



CO 



^ o 

CO cz 
5 -2 

CD O 

z < 



o 

CO 
CQ 



o 

CO 
CQ 



o 

CO 
CQ 



CO 

> 



CO 

> 

CO 

cz 

=) 

CO 



CO 

< 

CCD 
CO 

> 

CO 



OO 



CO 
CO 

> 

CO 



CD 



CO 
"co 



. , Invento«: Richard Meyer et alji Q Q 7 fc CJ 0 a Q&JL «T P Q S 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 12 of 37 



O 

O 
CO 
CD 

Q 



cz 
.O 

<: 



CO 
CD 

cz 



o 

a: 







O 


-2 










-o 




CD 


CD 


.== 




CO 


CO 


o. 


o. 


CD 


CD 






- — 


- i 






"O 




CZ 


cz 


CD 


CD 


CO 


CO 


_o 


_o 










CD 


CD 


E 


E 










"55 


"55 


CO 


CO 


oi 


CNJ 


"O 


"O 


CD 


CD 














"o 


"o 


cz 


CZ 


CO 


CO 














o 


_o 










■o 


TD 


CD 








CO 


CO 


o_ 


Q- 


CD 


CD 






"Mi 




"O 


"O 


CZ 


CZ 


CD 


CD 


CO 


CO 



CNJ 



CO 


CO 


Mi. 








configured" 


configured" 


"Mi 


"Mj 


send 


send 


o 


c> 


mer 


mer 




"~» — » 


Set 


Set 


oi 


csi 




il 


o 


,0 


R 

TD 
CD 


TZJ 
CD 


configui 


configui 






send 


send 


CD 


CD 


configui 


configui 


CZ 
CO 


CZ 
CO 











GO 



CD 

E 



CD 
CO 

CO 



CD 



CD 



O 
O 



CD 
CO 

■o 

CD 



CD 



O 
O 

"CJ 
CZ 
CO 



o 



CD 
CO 



O 
CD 



CD 
CO 

CO 



-o 

CD 



CD 



O 
O 



CD 
CO 

"O 

CD 

ZD 
CD 

cz 

8 

TD 
CZ 
CO 



CD 
CZ 

*E 
_(D 
"55 

TD 



CO 

CD 
CZ 

o 

"O 

CZ 
CD 



CO 
CO 

E 

CD 

E 
o 
o 

CD 
_Q 

CD 
CO 



CD 



CD 

o 

CO 



o 

CO 
JQ 



CD 
CD 



O 



CD 
CO 



CNJ 



O 
CO 

_o 



CD 

~cz 
O 

CNJ 
CO 

E 

"CZ 

Ql 
CD 



O 
O 
CD 
JZ3 



CNJ 



O 
CO 
-Q 



CD 
> 
CD 



O 
CD 



CD 
JZZ 

O 
oi 

CO 

E 
Ql 

CD 

E 

8 

CD 



O 
CD 



CD 
CO 



CD 

_cz 
O 

CNJ 
CO 

I 

CD 

E 
o 
o 

CD 
JQ 



O 
CO 



CD 



O 
CD 



CD 
CO 



CD 

ZD 
CD 



TD 
CD 



CO 



-92 

"55 



CO 
CD 



O 
"O 



CD 
-CZ 



CD 



CD 
CO 

CO 

E 
al 

CD 



O 
O 
CD 
-O 

CD 
CO 



CD 
-CZ 

o 

CO 



CD 



CO 



o 

CZ 

CD 
CO 



Q_ 
O 
CZ 

CD 
CO 



CD 



CD 

O 
CNJ 

CO 

E 
al 

CD 

E 
o 
o 

CD 



O 
CO 
CD 

CD 



O 
O 
CD 
-Q 



O 
CO 
OQ 
CD 

E 
o 
o 

CD 



o 



CD 
CO 



CNJ 
CO 



CM 
CO 



CD 

E 
o 
o 

CD 
-O 



CD 

E 

o 
o 

CD 



CD 
O 

oi 



o 

CO 



CD 
> 
CD 

O 
JZZ 

o 

CD 



TD 
CZ 

LU 

CO 

csi 

CD 
CD 

E 

CD 

CD 
CZ 

E 
Jd 
"55 



CD 
CO 

oi 

CD 



CO 
CO 
CD 

o 
p 



O 

-4— » 

CO 
!_ 

CO 



CD 



O 



CD 
CO 



O 
o 



CD 

o 

cvi 



o 
CO 



CD 

> 

CD 
O 

O 
CD 



CD 
CO 

csi 

_CD 

o 
a5 

-O 

E 

CD 



CD 



CD 
CO 

oi 

CD 
CZ 
CO 
CO 
CD 

o 
o 



CD 
-CZ 

O 
oi 

i — 
o 

CD 
CZ 

CO 
CO 

8 

O 



_o 

CO 

l_ 

CO 



CD 

"55 



CD 



Ol 
O 

CO 



CO 



CD 



CD 

szz 

o 

csi 

CD 
CZ 

CO 
CO 
CD 

o 
o 

Q_ 

Q_ 

O 

CO 
i 

CO 



,o 

CD 



CO 
CO 
CD 

o 

O 



O 
CD 



CO 
CO 
CD 
O 
O 



CO 



CD 



CO 



O 

CO 
i_ 

CO 



CD 



CD 
Q_ 



O 



CD 



CO 



Inventors: Richard Meyer et ^QQ 7^*3 Q fa „ Q J& A «3 O S 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 13 of 37 



Create Served 
Failover Set 



Creating Served 
Failover Set 



Served Failover 
Set Created 



Online 
Requested 




Transaction 
Complete 



Transaction 
Complete 



Failover Set 
Removed 



Deleting Served 
Failover Set 





Offline 
Requested 



Member 
Lost 



Removing Lost 
Component 



Transaction 
Complete 



Transitioning 
to Offline 



FIG. 14 Served Failover Set State Machine Diagram 



Inventors: Richard Meyer et alj_ 0Q ~? ig fj. 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 14 of 37 



External Management 
System(s) 



Commands & 
Polls 



A SNMP Traps 




Component State 
Change Events 



Sources for Failover Commands 



SW Image Management 



Management Service 





Failover 


Error 


Requests 


Analysis 


► 



Failover 
Requests 



Recovery 
Management 



Failover 
Commands 
>■ 



Failover 



Resource 
State Change 

Commands 
> 



Error 
Events 



Task 
Monitor 



Sources for Failover Error Events 



SP Software 
Entities 



RTOS and 
Drivers 



Diagnostics 



Heartbeat 



Services 
Framework 
Software 
Entities 



Resource 
Managers 



VSX Config. 
Service 



SP Software 
Entities 



Services 
Framework 
Software 
Entities 



1 



Hardware 
Errors 



Hardware 



FIG. 15 Fault Detection and Analysis Architecture 



Inventors: Richard Mq« et alr QO Xfe^i Q6 ^Hfc-WljiS 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 15 of 37 



Stepl 



ePPC 



PICO 



D6 



PPC c 



a1=PPC[v] 



\ 



\ 



ePPC 



PICO 



\ r- 
\ 

X! 



D6 



a1=PPC[v] 



Step 2 



PPC 



ePPC 



PICO 



r- 



a1=PPC[v] a2=ePPC[v] a3=PICO[v] 



r 



D6 



ePPC 



I v. 



— 

■T 

1 - ' " ^ 



PICO 



D6 



a1=PPC[v] a2=ePPC[v] a3=PICO[v] 



Step 3 

majority(a1,a2,a3) = majority(v,v,v) = v, No faults 
tf % o\ FIG. 16 No Faults 



Inventor: Richard Meyer et & p.Q & q Q fe m Q fe H C 3 Q £ 
Title: Fauover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 16 of 37 



Stepl 



ePPC 



PICO 



PPC 



a1=PPC[x] 



D6 



ePPC 



PICO 



a1=PPC[y] 



D6 



Step 2 



PPC 



ePPC 



PICO 



+ -------^-_ • _ 



/ 

i- 



a1=PPC[x] 



r 
* 



D6 



a2=0 



a3=0 



ePPC 



• r 



PICO 

- 1 



a1=PPC[y] 



D6 



a2=0 



a3=0 



Step 3 



majority(a1,a2,a3) = majority(x,0,0) = 0, transmitter fault 



FIG. 17 Transmitter fault (sends a bad value) 



Inventors: Richard Meyer et afe "QfJ "J? 1 > 
Title: Failover Processing In A Storage' System ' 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 17 of 37 





Step 3 



majority(a1,a2,a3) = majorityfOAO) = 0, transmitter fault 



FIG. 18 Transmitter fault (doesn't send a value) 



Inventors: Richard Meyer et ^ Q y.g, tgo fe , Q & J,«f-Q ^ 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 18 of 37 



Stepl 



ePPC 



PICO 



D6 



ppc - 



a1=PPC[v] 



ePPC 



PICO 



D6 



a1=PPC[v] 



Step 2 



ePPC 



PICO 



PPC 



r- f---^-.-,-^- ■ _ 



a1=PPC[v] a2=ePPC[y] a3=PICO[v] 



r 



D6 



ePPC 



• r - ' 



PICO 



D6 



a1=PPC[v] a2=ePPC[v] a3=PICO[v] 



Step 3 

;ou J majority(a1 ,a2,a3) = majorityCv.y.v) = v, Receiver fault 

FIG. 19 Receiver fault (relays wrong value) 



t„ P In , vent % s: Richard ™ e /z et loo *a o fe „ o a os 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 19 of 37 



Stepl 



PPC ^ 



ePPC 



a1=PPC[v] 



ePPC 



\ 

\ r- 
\ 



a1=PPC[v] 



PICO 



D6 



PICO 



D6 



Step 2 



PPC 



ePPC 



ePPC 



PICO 




PICO 




h Step 3 

' 1 ; majority(a1 ,a2,a3) = majority(v,0,v) = v, Receiver fault 



FIG. 20 Receiver fault (doesn't relay a value) 



Inventors: Richard Meyer et ab 0 QT & CgO to .li&I'^Or 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 20 of 37 



VSX1 
Backup 



vsc 




Alternate 






sFailover/ 





LC 






\Fai lover/ 





VSC 




Alternate 






vFai lover/ 





LC 






\Failover/ 







LC 


/^Ic^N 




\Fai lover/ 






vsxo 

Primary 




VSC 
Primary 






LC 


/^LC^X 




VFailover/ 





FIG. 21A Failover Service Architecture 



Inventors: Richard Meyer et iQ-nj fc CJ Q fe Q ig Q S 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 21 of 37 



Services 
QOOQ 




Services 




Services 


Framework 




Framework 




Framework 


► 


► 


■< 





OS 




OS 




OS 


Location 1 


Location 2 


Location N 



FIG. 21 B 



Persistent 
Data 




Failover Service 




Failover Service 



FIG. 22 An Arbiter for the Database 



Inventors: Richard Meyer et a^Q.Q y fe fi ^ q , 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 22 of 37 



Server 



Server 



Primary 
VSX 




pppppppp 

0 1 2 3 4 5 6 7 

nzirzziizzirz=irz=ic=:c=ic=i 
oooooooo 



' Optional ■ 

i Switch or ! 
Hub 



pppppppp 

0 1 2 3 4 5 6 7 

C=1[=D(Z=1IZZIIZ=IC=1I=DI=I 
OOOOOOOO 




Disk Array Disk Array 

FIG. 23 Shared Link 



Backup 
VSX 



Source 
(1) 



Message 
Passing 
Interface (1) 



SCSI 
Manager (1) 



SCSI 
Manager (2) 



Message 
Passing 
Interface (2) 



Destination 
(2) 



Send MSG from VSX (1) 
to VSX (2) 



I/O Request 



Send Diagnostics 
• ► 

XFER READY AI,ocate Buffer for Me N a 9 e 
M • 



One or More Data Frames 



Message Received 
(New Request) 



I/O Done 



End Response 
(Status) 



Deliver Status 
■< 



Deliver Message 
► 



F I G . 24 VSX to VSX Message Passing 



Inventors: Richard Meyer et aft ntl ~3F-aZ. £3 ^rn -r « «w 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 23 of 37 



Server 



Server 



■DDI 

ODDQDDO 



Primary 
VSX 



Management 
Server 



1=1 



00D0ODD 




oo oo 




pppppppp/ 


0 12 3 


4 5 6 7/ 


□ cziizzia 


□ □□a 


o o o o 


o o o o 



pppppppp 

1 2 3 4 5 6 7 

oooooooo 
























J 



Disk Array Disk Disk Array 

FIG. 25 Management Link 

Server Server 




Backup 
VSX 



Primary 
VSX 



pppppppp 

0 1 2 3 4 5 6 7 

i — ii — i mi — ii — nzzimcj 
o o o o o o o o\ 




Switch or i 
__H_ub__ 

Sois 




PPPPPPPP 
0 1 2 3 4 5 6 7 







□ 
□ 

























Disk Array LJ Disk Disk Array 

FIG. 26 Shared Disk 



Backup 
VSX 



Inventor: Richanl Meyer etal^qO"?" &<30& tt OiteJ.^? 
Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 24 of 37 



Source 
(1) 



Message 
Passing 
Interface (1) 



SCSI 
Manager (1) 



SCSI 
Manager (2) 



Message 
Passing 
Interface (2) 



Destination 
(2) 



Send MSG from VSX(1) 
to VSX (2) 



I/O Request 



Write 



Buffer 



End Response 
(Status) 



I/O Done 



Get Message 
(Polls) 



I/O request 



Read Buffer 



End Response 
(Status) 



I/O Done 



Deliver Message 
• 1 ► 



FIG. 27 VSX to VSX Communication Using Shared Disk 



™ B f T n T Ricl r d , M ^ et 7 ti, «§.a €c . B & .1. «Ti Q 2 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 25 of 37 



Server 



Primary 
VSX 



pppppppp 

0 1 2 3 4 5 6 7 



CZI o □ CZl CTD CZI C5 

o o o o 



Server 




Backup 
VSX 



pppppppp 

0 1 2 3 4 5 6 7 




Failover Set (HA) 



Member Composition 



Member Unit 



I 



VSX 



Member Unit 



VSX 




Disk Array 



Server 



Disk Array 

FIG. 28 2 Node HA Configuration 

Server 




oo oo 

PPPPPPPP 


0 12 3 


4 5 6 7 


nnnn 
o o o o 


o o o o 


oo oo 

pppppppp 


0 12 3 

CZI CZ1 CZZ1 

o o o o 


4 5 6 7/ 

□ □ czj □ 
o o o o 



Backup 



GigE 
Switch 



Alternate 



oo oo 

pppppppp 

0 1 2 3 4 5 6 7 

-C=3 1=3 C3 £=3 tm CT3 CT3 CD 



o o o o o 



oo oo 

pppppppp 

^X.0 1 2 3 4 5 6 7 

□ izziczicziizDcncDCD 



oooooooo 




Alternate 



Failover Set (HA) 



Member Composition 



Member Unit 



Member Unit 



Failover 
Set (FT) 



Composition 



Member 

u 



VSC 



Member 

T 



Failover 
Set (FT) 



Composition 



Member 



Member 



VSC 



VSC 



VSC 

























































r 



Disk Array Disk Array 

FIG. 29 Hierarchical HA Configuration 



. I"ve"tors: Richard Meyer et 4QQ y fc, fe ,061902 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 26 of 37 















CD 








_o 








E 


'a 






CD 


=> 


















(VH) 


ositio 


mber 


Unit 






CD 


Se 


E 






lover 


berCi 


CD 

E 


» 

"c 


TO 


E 


CD 


ZD 


U_ 


CD 
















Member 


Unit 



X 



X 
CO 



X 
CO 



X 
CO 




oTx 
E co 



Q_ 


^0° 


a_ 




Q_ 




Q_ 


^-Qo 


Q_ 


coQo 


a. 


cmQo 


o CL 


-Do 


So- 




o 



CO 
CD 

-o 

O 



O 
CO 

CD 



Inventors: Richard Meyer et aljJ.Q y , M ^ Q4& J_4gc 0 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 27 of 37 



CD 



CD 



X 
CO 



CD 
CO 

CD 
> 



CO 

o 
E 

O v 

O v 
i — 

CD 
_Q 

E 

CD 



CD 
-O 



CD 



CD 



CD 



X 
CO 



X 
CO 



CD 

E 

CD 



X 
CO 

> 







Switch 


111 


_ _ 




M 




CO 
CD 

O 
I 



CO 

CD 



Inventors: Richard Meyer et aJj-Q-Q 0l& w .Qf^'A 9f! B 

Title: Failover Processing In A Storage System 
Patent App. No. 10/076,906; Filed: 2/13/02 
Charles Hamilton (650-326-2400) Docket No. 20949P-000800US Page 28 of 37 




1 . VSC Crashes (Host Processor) 

2. Rest of system detects VSC crash 

3. Error Analysis determines Member fails, which translates into a "Primary Lost" event 

4. Activate JCP in Master mode and enable the virtual services, Stop Ports on failed Primary 

5. Reset affected devices, Cleanup reservations and locks, Set Unit Attention 

6. Restart management requests 
v 7> Restart RCON and FORMAT 

\ FIG. 32 VSX Failover, Primary Fails 
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1. LC Crashes (Host Processor) 

2. Rest of system detects LC crash 

3. Error Analysis determines 10 Path fails for all devices (server and storage) on LC 

4. Upstream hLUNs report CHECK CONDITION for all devices connected to ports on failed LC. 
RCON and FORMAT aborted, if necessary. 

5. Restart RCON and FORMAT, if necessary 
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1. FC ASIC Crashes 

2. LC detects FC ASIC crash 

3. Error Analysis determines 10 Path fails for all devices (server or storage) on FC ASIC 

4. Upstream hLUNs report CHECK CONDITION for all devices connected to failed FC Ports. 
...^ RCON and FORMAT aborted, if necessary. 

1 5> Restart RCON and FORMAT, if necessary 



FIG. 34 10 Path Failover - FC Port Fails 
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1. Link down on port 

2. LC detects FC Port link down 

3. Error Analysis determines IO Path fails for all devices (server or storage) on FC Port 

4. Upstream hLUNs report CHECK CONDITION for all devices connected to FC Port. 
;,. RCON and FORMAT aborted, if necessary. 

5. Restart RCON and FORMAT, if necessary 



FIG. 35 IO Path Failover - Link Down 
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